Functional and Nonfunctional Testing – Stakeholder Management

Performance Testing Software Before the Launch aka Functional and Nonfunctional Testing

I am beginning to realise that nonfunctional testing particularly among business folks is a rather murky area that is worth discussing. Even testers may have a hard time understanding why we need to spend quite some time and effort on various aspect of performance testing. And it’s doubly hard with business stakeholders.

So here’s my experience with some of the things you need to discuss with C-suite, product people, infra and ops people and your own developers. It takes longer than expected. You are likely to face more resistance to your best intentions, active and passive, than you expect. And you need to educate the people around you so they have a better, deeper understanding of the implications of not doing performance testing. You need to be very clear about the kinds of risks the business takes if they neglect this area and the damages it may have for the product, and the implications for the total cost of ownership for the service you run. This is a long process, so start early and focus on the long term goals.

There are two scenarios where you may be right now:

First, if the product is still looking for a few real customers, it’s really hard to tell what is going to break. Second, your product is already out in the wild, and there are support tickets and customer complaints, so the need to do some performance tuning is blatantly obvious. So how do you handle the first situation and how do you handle the second one when you are a tester with nonfunctional ideas on their mind. (And the off-by-1-error scenario is you are already past all of this and have a performance gig well under way. If so, email me and let’s compare strategies to see where we could improve further.)

Nonfunctional Testing for Software Products before the Launch (The Why, What and How)

The first scenario requires a rather mature organisation to realise that the first version of the software is probably not good enough for the market. This means that you need to remind people that nonfunctional aspects of even an MVP must be tested properly, not just be assumed to work.

Most people assume that the product or service is going to be okay, might be a tad slow, but customers will love it after all. But most often it’s not the speed, or rather the slowness that is the issue. Your software is going to crash, instead. Not under the heavy load, though. If you are lucky, it’s more likely to be operational issues. Logs are not rotated, node runs out of disk on week 7 and the whole thing comes to a screeching halt and because there is no proper monitoring on it, nobody will know until Monday when the customer takes a look.

You could say, “well, it’s an operational thing, so it is none of my concern, I run a development team here, not an ops team”. But that would be wrong. Performance is an operational issue. Your software exists in an ecosystem on a multilayer platform, so everything that has an impact on its performance is your concern. Your customer does not care why it failed, for them the only thing to remember is that it did.

If you are not so lucky, and say you have Java (or Python or C#) on your hands, you might end up with a nasty memory leak that only manifest during lunar eclipses if the customer has a first and a middle name, both starting with X. And it takes weeks to figure out where it’s bleeding. And weeks to fix it, test it and patch it.

So how do you prepare for these as a tester?

Before going into production, run the following three sets of performance experiments. But before you run these test talk to a lot of people to explain why they are needed and what prerequisites you need.

• – Soak test your application for three weeks.
• – Restart all components a fair few times.
• – Put half a million records in your DB and do a regression test under moderate load.

Why test all these things?

Why go through all these hoops you already know that your features are working right? Because you know how it works, but you don’t know how it fails. Because you don’t yet know how your software, your whole solution and your infrastructure fails. So here are ten reasons why you should do an initial set of performance tests on your shiny new product.

Look at the list below as the major agenda items in all your discussions with all the business people, from sales to teas leads and the CEO.

Top 10 Reasons to Convince Your CEO / Head of Product to Performance Test Before the Launch

#1 Your first customer is really hard to get. There will be a little celebration on the day they sign the contract and another they go into prod. So the last thing you want is a major incident a few days later because you never ran your own software for more than a few hours. You don’t want to lose the good will you worked for, for months, in 4 days because of a silly little glitch. No amount of explaining will make that go away.

#2 The soak will ensure there are no major robustness glitches. It will help you identify software bugs you did not know were there. It will let you roll out a software that is good enough, not in your eyes, but in the eyes of your first customer.

#3 Restarting all components will help you learn how your system recovers, or hangs for that matter. For functional tests, your system was already running smooth, chugging along the tracks nicely. For production, you need to know how long it takes to start up after a glitch. There is no question that one or more of your components will go down. The questions is not why they did – you will have plenty of time to figure that out later. For now, the question is, can it recover, can we speed up the recovery, can we put in some self healing mechanisms?

#4 Half a mill records in your DB sound like a lot, but that is the bare minimum. So far you have only driven this ride with only you sitting at the wheel. And trust me, it feels a whole lot different when my mum sits in the passenger seat, man, that small car of mine feels like a completely different car altogether.

#5 You need to train your team to be able to start up your system at 2am on a public holiday from a mobile phone. If the first time they are doing it is during a customer incident on a production system, it is going to be unnecessarily stressful and unduly long and complicated. By the time you are done with the second exercise, you have the wiki pages needed for emergency startups and restarts.

#6 At this point in time, you have assumptions about which components might have interdependencies. But a lot of this coupling is invisible until you simulate crash like scenarios. So do yourself a favour by running through a few restart experiments to make sure that you know how to run your software.

#7 When your software is ready for production, your operations team is not. Starting up your software and running it for a few weeks will help you train your team to set up the production environment right.

#8 If your database needs some tweaking, it should come out when you run your regular functional test on a loaded DB. You might need another table, view some other change in the structure. And that is way easier, faster, cheaper and safer to do before you go into production than after.

#9 Running regression with some other traffic also helps you uncover race conditions and corner cases that you missed and would only come out in prod when there is some traffic on the system already.

#10 Introducing nonfunctional testing to your organisation early in the SDLC help you in many ways.

– It positions you as someone who cares about the overall quality of the software. It is the quality championing that all the job ads are looking for.

– It helps people understand that a customer mindset is more than just wanting all the features and functionality. Customers have an unspoken understanding that your software will just work. That is will not be a constant source of pain and headaches, escalations and disappointments. They just want it to run smoothly and without a glitch most of the time, and it is your job to make sure it does.

– It also helps you pave the way for the long run for more nonfunctional goodies, from volume and load testing to recovery testing and the technical aspects of disaster recovery.

+1 aka #11

These are also fairly cheap to set up and run. It would add another six weeks or so to the delivery timeline. No major hardware or software investment is needed beyond what you would need to have anyway. You can skill up as you go along even if there’s something new in the perf tooling. Low risk, low cost, completely predictable, and it lets you roll out a product that is a lot more stable and resilient.

The following section describes what to test as part of your prelaunch performance testing, how and why, with some explanations on how to interpret the results of your performance testing experiments.